Method for operating user functions based on eye tracking and mobile device adapted thereto

ABSTRACT

An eye tracking based user function controlling method and a mobile device adapted thereto are provided. A camera unit of a mobile device is activated while a specific user function is executed. A gaze angle of a user&#39;s eye is acquired from an image obtained via the camera unit. An eye tracking function is executed in which execution state is controlled according to the gaze angle.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.13/681,719 filed on Nov. 20, 2012 which claims the benefit under 35U.S.C. §119(a) of a Korean patent application filed on Nov. 29, 2011 inthe Korean Intellectual Property Office and assigned Serial No.10-2011-0125911, the entire disclosure of which is hereby incorporatedby reference.

BACKGROUND

1. Technical Field

This disclosure relates generally to controlling user functions andconserving power in electronic devices, and in particular, in mobilecommunication devices.

2. Description of the Related Art

Mobile devices such as smart phones and tablet PCs have become widelypopular due to their portability and support of a wide variety of userfunctions. Recent designs provide integrated user functions, such as acamera function in conjunction with storing acquired photographs andvideos, immediately searching for the acquired photographs or videos,etc. As newer models of mobile devices are equipped with relativelylarge displays, although they're still easily hand carried, a tradeoffin user controllability has arisen. For example, a notebook sized tabletcan be held by one hand while the user touches the display with the samehand or the other hand; however, he/she may be unable to continueholding it stably due to the weight and size. This may cause a varietyof problems, such as the user losing his/her grip of the mobile device,or causing a muscle injury.

To conserve battery power, conventional mobile devices change state froma normal state, in which the display has normal brightness, to a powersave state in which the display is dimmed or shut off (the latter isreferred to herein as a sleep state). The transition to the power savestate is made after a predetermined elapsed time of not receiving anyuser input, such as a touch input command. This elapsed time isessentially a predicted time where the user is no longer activelyviewing the display. To this end, a state control function is set underconditions set by default or by preference settings.

However, the set conditions may not always match a particular user'sinteraction with the mobile device. For example, the predeterminedelapsed time that triggers the transition from normal mode to sleep modeis a fixed value, not adaptively varied according to the user's state orthe environment. Therefore, the conventional mobile device may enter apower save or sleep mode at an undesirable time point, e.g. when theuser still wants to use the device. In this case, the user must apply anadditional operation to the mobile device, e.g., an awaking operation.The additional operation inconveniences mobile device users, and inparticular, users of devices with larger displays which tend to be moreactively utilized. Furthermore, if the elapsed time for transitioning toa power save mode is set too long, battery power is essentially wastedin the time period after the user stops viewing the display but prior tothe elapsed time point.

SUMMARY

Disclosed herein is a method that can control efficiently and adaptivelyuser functions of a mobile device, based on eye tracking, and a mobiledevice adapted to the method.

In accordance with an exemplary embodiment of the invention, a mobiledevice that controls user functions based on eye tracking is provided,including: a camera unit that acquires images of a subject; a controllerconfigured to acquire a gaze angle of a user's eye from an imageacquired via the camera unit, and to execute an eye tracking function inwhich an execution state of a user function is controlled according tothe gaze angle; and a storage unit that stores data corresponding to theuser function, and a program to track gaze angles of a user's eye.

Also provided is a method for operating user functions in a mobiledevice based on eye tracking. A camera unit of the mobile device isactivated while a specific user function is executed. A gaze angle of auser's eye is acquired from an image obtained via the camera unit. Aneye tracking function is executed in which execution state is controlledaccording to the gaze angle.

BRIEF DESCRIPTION OF THE DRAWINGS

The features and advantages of the invention will become more apparentfrom the following detailed description viewed in conjunction with theaccompanying drawings, in which:

FIG. 1 illustrates a schematic block diagram of a mobile device with aneye tracking function, according to an embodiment of the invention;

FIG. 2 illustrates an exemplary configuration of the controller shown inFIG. 1;

FIG. 3 illustrates user interface screens that support user functionsvia eye tracking, according to an embodiment of the invention;

FIG. 4 illustrates user interface screens that support user functionsvia eye tracking, according to another embodiment of the invention;

FIG. 5 illustrates user interface screens that support user functionsvia eye tracking, according to a further embodiment of the invention;

FIG. 6 illustrates a flow chart that describes a method for controllinga user function based on eye tracking, according to an embodiment of theinvention; and

FIG. 7 illustrates screens to describe a method for making a call basedon eye tracking, according to an embodiment of the invention.

DETAILED DESCRIPTION

Hereinafter, exemplary embodiments of the invention are described indetail with reference to the accompanying drawings. For the purposes ofclarity and simplicity, detailed descriptions of well-known functionsand structures incorporated herein may be omitted to avoid obscuring thesubject matter of the invention. A detailed description ofconfigurations that includes the same elements and performs the samefunctions will not be explained repeatedly in the description.

Although the drawings represent embodiments of the invention, thedrawings are not necessarily to scale and certain features may beexaggerated or omitted in order to better illustrate and explain theinvention. It should be understood that the invention is not limited tothe drawing scale.

It is to be understood that the singular forms “a”, “an” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

FIG. 1 illustrates a schematic block diagram of a mobile device 100 withan eye tracking function, according to an embodiment of the invention. Amobile device according to the invention can be any informationcommunication device, multimedia device, etc., operated according tocommunication protocols corresponding to various types of communicationsystems. Examples of a mobile device include mobile communicationterminals, Portable Multimedia Players (PMPs), digital broadcastplayers, Personal Digital Assistants (PDAs), audio players (e.g., MP3players), mobile game players, smart phones, laptop computers, hand-heldPC, etc.

Mobile device 100 includes a communication unit 110, an input unit 120,an audio processing unit 130, a display unit 140, a storage unit 150, acamera unit 170 and a controller 160.

When a specific user function is executed or a specific state isactivated in the mobile device 100, the mobile device 100 automaticallyactivates the camera unit 170 and controls it to acquire images at acertain angle. The mobile device 100 measures the motion of a user'seyes using the acquired images, via eye tracking, and identifies thedirection of the user's gaze. If the direction of the user's gaze is apreset direction, the mobile device 100 generates a commandcorresponding thereto. After that, the mobile device 100 alters ormaintains the current state or executes a function, according to thegenerated command. Since the mobile device 100 can be controlledaccording to the user's gaze, the user can manage operations easily,adaptively, and effectively on the mobile device 100.

The communication unit 110 supports communication of the mobile device100, and is implemented with a module for supporting communication, inparticular, mobile communication. The communication unit 110 establishescommunication channels with a mobile communication system andtransmits/receives signals thereto/therefrom. For example, thecommunication unit 110 establishes at least one of voice, video, anddata service channels with a mobile communication system, andtransmits/receives corresponding signals via the channels. (It is notedhere, that if mobile device 100 is implemented without supporting acommunication function, the communication unit 110 can be omitted.)

The communication unit 110 may also perform operations associated withthe camera unit 170. For example, if the mobile device 100 receives, viathe communication unit 110, a call connection request message or amessage replying to the reception of a service message, from anothermobile device, the controller 160 outputs information indicating themessage has been received to the display unit 140 and/or the audioprocessing unit 130. During this process, the controller 160 maytransfer the signal corresponding to the message reception event fromthe communication unit 110 to the camera unit 170. That is, when eyetracking mode is activated, if the communication unit 110 receives amessage, the camera unit 170 is activated for a preset period of timeand takes a video at a preset angle, which is used to support eyetracking. The controller 160 detects the user's eyes and the eye anglesfor eye tracking from the video acquired via the camera unit 170, andcontrols the communication unit 110 based on the detection. That is, anexemplary eye tracking mode can be established for facilitatingacceptance of incoming calls. In this mode, when an incoming call isreceived, if the user gazes at the mobile device 100 display, a callconnection request message is automatically transmitted, replying to theincoming call via the communication unit 110, thereby establishing thecall connection therebetween. Thus if the mobile device 100 receives acall connection request message from the other mobile device, it canestablish the call connection with the other mobile device automaticallyif the user gazes the mobile device 100, without requiring an additionaloperation. In another embodiment, an automatic call connection of themobile device 100 with the other mobile device is implemented via facerecognition (identification), in such a way that if the mobile device100 identifies a predefined face, it performs the automatic callconnecting process.

The input unit 120 generates signals required for the operations of themobile device 100. The input unit 120 may be implemented with akeyboard, a keypad, key buttons, etc. according to the compatibilitywith the mobile device. If the display unit 140 is implemented with atouch screen, the touch screen performs at least part of the functionsof the input unit 120 (although touch panel 143 is shown as part ofdisplay unit 140). In an embodiment of the invention, the input unit 120generates input signals for supporting an eye tracking functionaccording to a user's command, for example, signals for executing orreleasing the eye tracking mode, signals for calling user functionsassociated with the eye tracking function, signals for associating aneye tracking function with a user function, etc. The input unit 120transfers the generated input signals to the controller 160, and thecontroller 160 executes corresponding functions according to the inputsignals.

The audio processing unit 130 outputs audio signals that are set duringthe operations of the mobile device 100, generated when audio filesstored in the storage unit 150 are played back, or received from theoutside. The audio processing unit 130 also supports an audio signalcollecting function. To this end, the audio processing unit 130 includesa speaker (SPK) and a microphone (MIC). In an embodiment of theinvention, when a user function set with eye tracking is initiallyactivated, the audio processing unit 130 outputs the corresponding audiosignal indicating that the eye tracking function is being executed. Ifthe mobile device 100 that can support an eye tracking function has notexecuted the eye tracking function, the audio processing unit 130 mayoutput an audio signal informing the user that a corresponding eyetracking function may be set. Note that an audio signal is only outputat the time the tracking function is initially set. This audio signaloutputting function may be disabled according to the user's settings.

The time point that the audio processing unit 130 outputs an audiosignal may be controlled via eye tracking. For example, the audioprocessing unit 130 may be controlled in such a way that it outputs anaudio signal for a currently executed function only if the camera unit170 acquires a user's gaze angle to the mobile device 100, correspondingto a preset angle; however it stops outputting the audio signal if theuser gazes at the mobile device 100 at an angle other than the presetgaze angle. Herein, reference to a “gaze angle” signifies a gaze anglewith a tolerance range, as will be understood by those skilled in theart. Thus, an angle other than a preset gaze angle means an angleoutside the predetermined tolerance range of the preset gaze angle. Inother words, “gaze angle” should be understood to mean “gaze anglerange” when the application so dictates.

The display unit 140 provides a variety of screen interfaces requiredfor the operations of the mobile device 100. For example, the displayunit 140 supports an idle screen, menu screens, etc. In an embodiment ofthe invention, the display unit 140 provides a screen according to theexecution of a user function associated with eye tracking, a settingscreen for associating a user function with an eye tracking function, ascreen for enabling or disabling an eye tracking function, etc.

The display unit 140 may be implemented with a touch screen. In thiscase, the touch screen includes a display panel 141 and a touch panel143, where the touch panel 143 is installed at the front of the displaypanel 141 (hereinbelow referred to as just “the display” or “the screen”for brevity). The display panel 141 displays images, text, etc.,corresponding to the variety of screens described above. The touch panel143 includes a touch effective area and a non-touch effective area,defined according to the features of the screens displayed on thedisplay panel 141, and transfers a signal corresponding to a touch eventthat occurs on the touch effective area to the controller 160. The touchpanel 143 may be activated via eye tracking. For example, if the userviews the mobile device 100 at a preset gaze angle, the touch panel 143is activated and detects the presence of a touch event according tohis/her touch action. On the other hand, if the user views the mobiledevice 100 at an angle other than the preset gaze angle, the touch panel143 is disabled. The mobile device 100 may be implemented in such a waythat it supports a temporary lock function or unlock function via eyetracking.

The storage unit 150 stores an operation system, application programsimplementing algorithms, data, etc., required for the operation of themobile device 100. In an embodiment of the invention, the storage unit150 stores an eye tracking supporting program 151 for supporting eyetracking functions, and an eye tracking algorithm 153 for tracking agaze angle of the user's eye in order to support the eye trackingsupporting program 151.

In exemplary implementations, the eye tracking supporting program 151includes a number of routines related to the operation of eye trackingfunctions as follows: a routine for activating the camera unit 170 if auser function associated with eye tracking is executed; a routine foracquiring images via the activated camera unit 170; a routine forrecognizing the face and eyes from the acquired images; a routine foridentifying the gaze angle of the eyes based on the recognition result;and a routine for controlling a currently executed user functionaccording to the identified gaze angle. “User functions” as used hereinrefer to a variety of functions executed in the mobile device 100, viathe components, e.g., the communication unit 110, display unit 140,audio processing unit 130, etc. Examples of user functions are a callfunction, a content file playback function, a file search function, aweb access function, a function for outputting an idle screen, menuscreens, etc., a broadcast receiving function, etc.

The eye tracking algorithm 153 analyzes images acquired by the cameraunit 170 activated via the eye tracking supporting program 151. The eyetracking algorithm 153 can include both a face recognition algorithm andan iris recognition algorithm. Here, “face recognition” refers to adetection of the presence of any imaged face in the field of view ofcamera unit 170, not necessarily a particular person's face. That is, anidentity detection is not required for the face recognition of theembodiments herein. Briefly, the eye tracking supporting program mayoperate as follows: when the camera unit 170 acquires an image, theprogram recognizes the outline of a face in the acquired image via theface recognition algorithm of the eye tracking algorithm 153. The eyeportion is then extracted from the recognized face image, and theninformation is acquired regarding the gaze angle of the eyes from theeye portion. The program compares the acquired gaze angle withpredefined information, and supports the state alteration of a currentlyexecuted user function, which will be described in detail laterreferring to the accompanying drawings.

The camera unit 170 is activated and acquires images, under the controlof the controller 160. If the mobile device 100 is set in such a waythat a user function associated with eye tracking is activated or an eyetracking function is activated by default, the camera unit 170 isactivated and acquires images when the mobile device 100 is activated.The camera unit 170 transfers the acquired images to the controller 160,and the controller 160 analyzes them according to the eye trackingfunction. The acquired images refer to still images acquiredperiodically at regular intervals or a video acquired in real time. Ifthe execution of a user function associated with an eye trackingfunction is terminated, the camera unit 170 may be automaticallydisabled (until activated by the user in a camera mode).

The controller 160 controls the flow of signals, the informationcollection, and the output operation, in order to support the eyetracking function associated with images acquired by the camera unit170. To this end, the controller 160 supports an image collectingfunction and an information analyzing function, and controls userfunctions via these two functions.

FIG. 2 illustrates a functional block diagram of an exemplary controller160 shown in FIG. 1. Controller 160 includes a camera informationcollecting unit 161, an eye tracking determination unit 163, and afunction operating unit 165.

The camera information collecting unit 161 determines, if a specificuser function is activated, whether the user function is associated withan eye tracking function; if so, the controller activates the cameraunit 170. The camera information collecting unit 161 sets the cameraunit 170 according to preset values and controls it to acquire images.The camera information collecting unit 161 executes the face recognitionprocess with respect to the user's face and allows the camera unit 170to automatically focus on the iris. After performing the automaticfocusing process, the camera information collecting unit 161 acquiresthe images of the user's face with the eyes, i.e., irises, and transfersthem to the eye tracking determination unit 163. The mobile device 100stores a list of user functions associated with eye tracking in thestorage unit 150 in order to determine whether a user functionassociated with eye tracking is activated. If a specific user functionis activated, the camera information collecting unit 161 identifies itin the list and activates the camera unit 170 to support it via eyetracking.

The eye tracking determination unit 163 analyzes an image transferredfrom the camera information collecting unit 161 and the gaze angle ofthe eye therein and transfers the analysis result to the functionoperating unit 165. To this end, the eye tracking determination unit 163may use the eye tracking algorithm 153 stored in the storage unit 150.The eye tracking determination unit 163 first performs the facerecognition in the received image and extracts the eye portion from theface image. The eye tracking determination unit 163 identifies the gazeangle of the eye from the eye portion. For example, in some embodiments,when the position and orientation of the user's eyes relative to thefront surface of the display are such that the user is determined to begazing at any point on the screen, the gaze angle can be considered zerodegrees. In other applications, such as an e-book reader applicationdescribed in connection with FIG. 5, the gaze angle is analyzed todetermine the point on the screen at which the user is currently gazing.

A gaze angle, which can also be considered a gaze “direction,” can alsobe detected via the algorithm as equaling zero degrees when the user'sface is determined to be substantially in front of the display, and thegaze direction is detected as parallel to a normal to the front surfaceof the display. A gaze angle of zero degrees can also be detected viathe algorithm if the user's face is substantially centered in an imageand the image of the user's irises is substantially centered in theimaged eyes. A zero degree gaze angle can also be detected if the user'shead is offset from the front of the display, but the user is directlygazing at the front of the display by viewing it from the corners of hereyes.

A non-zero degree gaze angle could be determined automatically if theuser's face is no longer detected, e.g., the user has put down themobile device or walked away from it.

With the gaze angle determined, the eye tracking determination unit 163can, in at least some embodiments, determine the point on the displayunit 140 where the user gazes (or if the user is no longer gazing),based on the identified gaze angle. In addition, the eye trackingdetermination unit 163 transfers the identified gaze angle to thefunction operating unit 165.

The function operating unit 165 determines whether to alter or maintainthe state of a currently executed user function based on the informationregarding the identified gaze angle transferred from the eye trackingdetermination unit 163. To this end, the function operating unit 165identifies a currently executed user function and generates a commandaccording to the gaze angle information. The function operating unit 165can control a user function according to the generated command. In anembodiment of the invention, the eye tracking function may be applied tothe mobile device 100, periodically or in real time. Therefore, thefunction operating unit 165 may alter or maintain a user functioncorresponding to a gaze angle that is periodically received. If thefunction operating unit 165 receives a gaze angle that differs from thepreviously received value—i.e., a newly detected gaze angle is outside apredetermined range associated with the performance of the currentlyexecuted user function, e.g., a range of zero+/−X degrees—it changes thecurrent user function to another user function. If the functionoperating unit 165 receives the same gaze angle as the previouslyreceived value, it maintains the user function.

As described above, the mobile device 100 according to the invention cancontrol, while a specific user function is being executed, the functioneffectively and adaptively via eye tracking, without operating anadditional touch action or an input button. A detailed description ofthe mobile device 100 for controlling user functions based on eyetracking is described as follows referring to the accompanying drawings.

FIG. 3 illustrates user interface screens that support user functionsvia eye tracking, according to an embodiment of the invention. In thisembodiment, the mobile device 100 alters the mode where a user functionis executed to a power save mode, e.g., a low illumination mode that maybe followed by a sleep mode (display turned off), via eye tracking.

Referring to FIG. 3, if the user operates the mobile device 100 toactivate a specific user function, e.g., a content view function, itactivates an illumination function of the display unit 140. To this end,the mobile device 100 supplies, if the display unit 140 includes abacklight, electric power to the backlight, whereby it emits a certainlevel of light. Alternatively, if the display unit 140 includes OLEDs,the mobile device 100 supplies electric power to them, controllingemission of a certain level of light. In conjunction with the backgroundillumination, the mobile device 100 retrieves the user's selectedcontent and displays it on the display panel 141 as shown in diagram 301of FIG. 3.

If the mobile device 100 ascertains that the content view function isassociated with eye tracking, it activates the camera unit 170 andrecognizes the iris of the user's eye. In diagram 301, a user's iris isillustrated pointing in a direction normal to the front surface of thedisplay, coinciding with a gaze angle of zero degrees as discussedearlier. Thereby, it is ascertained that the user is gazing at thedisplay. Hereafter, a gaze angle of zero degrees (i.e., zero degrees+/−xdegrees, where x is predetermined) will be used as an example of adetection that the user is gazing at the display. A non-zero gaze anglewill be referred to as a detected angle in which the user is not gazingat the display, or the presence of the user's face is no longerdetected. A non-zero gaze angle will also be referred to as a “firstgaze angle”. If the mobile device 100 ascertains that the user doesn'tgaze at the display unit 140 via the acquired gaze angle, it alters thelevels of light, step by step, as shown in diagram 303, 305, and 307.That is, if the mobile device 100 identifies a non-zero gaze angle ofthe user's eye, meaning that the user doesn't gaze at the display unit140, it controls the display unit 140 to emit a certain level of lightfor a first period of time as shown in diagram 303. The level of lightin diagram 303 may be lower than the level in diagram 301. After that,if the mobile device 100 ascertains that the non-zero gaze angle ismaintained for the first period of time, i.e., the user has not gazed atthe display unit 140 for the first period of time, it controls thedisplay unit 140 to reduce the amount of emitted light to a lower levelthan the previous level as shown in diagram 305. After that, if themobile device 100 ascertains that the non-zero gaze angle is stillmaintained for a second period of time greater than the first period oftime, i.e., the user has not gazed the display unit 140 for the secondperiod of time, it controls the display unit 140 to turn off theillumination function as shown in diagram 307. If the mobile device 100acquires the non-zero gaze angle for a certain period of time afterturning off the illumination function, it may operate automatically in asleep mode.

On the contrary, if the mobile device 100 identifies a zero degree gazeangle of the user's eye (hereafter, “second gaze angle”), meaning thatthe user gazes at the display unit 140, at a time point that it displaysthe screen as shown in diagram 305 or 307, it returns to the operationstate where the display unit 140 displays the screen in the originallevel of light as shown in diagram 303. However, if the mobile device100 identifies the second gaze angle of the user's eye in a certainperiod of time from a time point that it entered a sleep mode (as shownin diagram 307), it may display a lock screen for releasing the lockedscreen, to which the mobile device 100 automatically locks the screenwhen entering the sleep mode according to a preset routine. To this end,the mobile device 100 needs to activate the camera unit 170 to supporteye tracking, for a preset time period, in a state where the displayunit 140 is turned off as shown in diagram 307, a state where the mobiledevice 100 enters a sleep mode, or a state where a certain period oftime has elapsed from a time point that the mobile device 100 entered asleep mode. In addition, the mobile device 100 may also stop eyetracking according to the state where the display unit 140 is turned offas shown in diagram 307, the state where the mobile device 100 enters asleep mode, or the state where a certain period of time has elapsed froma time point that the mobile device 100 entered a sleep mode. That is,if the display unit 140 is turned off, the mobile device 100 enters asleep mode, or a certain period of time has elapsed from a time pointthat the mobile device 100 entered a sleep mode; the mobile device 100may stop eye tracking and disable the camera unit 170.

Meanwhile, if the user continues to gaze at the display unit 140 whiledisplaying a screen as shown in diagram 301, i.e., the mobile device 100acquires the second gaze angle, the mobile device 100 controls thedisplay unit 140 to maintain the same state showing the screens as shownin diagrams 302, 304, and 306. The mobile device 100 may not execute amode where the current state is automatically altered to another stateuntil a certain condition is satisfied or until it receives an inputsignal in a preset period of time. Therefore, although the user has readthe content on the display unit 140 for a relatively long period of timewithout generating an input signal, the mobile device 100 does not entera sleep mode or reduce the level of light. This allows the user toconveniently use the mobile device 100 while executing the content viewfunction. By contrast, in conventional devices, a display dimming/sleepmode is automatically entered if no new input command is received withina predetermined time period, even if the user continues to read acurrent content screen. This annoyance is eliminated with presentlydescribed embodiment of FIG. 3.

FIG. 4 illustrates user interface screens that support user functionsvia eye tracking, according to another embodiment of the invention. Inthis embodiment, the mobile device 100 controls the content playbackfunction via eye tracking.

Referring to FIG. 4, the user operates the mobile device 100 to executea content playback function. To this end, the user selects thecorresponding content in the storage unit 150, and operates the mobiledevice 100 to generate a command for playing it back. For example, ifthe user requests to play back a video, the mobile device 100 calls anapplication program for playing it back and controls the display unit140 to display a screen according to the playback as shown in diagram401.

The mobile device 100 identifies whether the content playback functionis associated with eye tracking. If so, it activates the camera unit 170and acquires images periodically or in real time, in order to supportthe eye tracking function (which is assumed for diagram 401).

If the mobile device 100 acquires the first (non-zero) gaze angle of theuser's eye, meaning that the user does not gaze at the display unit 140,while executing the eye tracking function, it enters a mode where itdisplays a screen as shown in diagram 403 and pauses the contentplayback. For instance, if the mobile device 100 acquires the first gazeangle of the user's eye immediately since the user does not gaze at thedisplay unit 140, it enters a mode where it pauses the content playback,as shown in diagram 403. To this end, the mobile device 100 executes theeye tracking function in real time. Alternatively, the pause operationis implemented only if the mobile device 100 acquires the first gazeangle for more than a predetermined minimum period of time.

If the user gazes the display unit 140 in a state where it displays thescreen as shown in diagram 403, i.e., if the mobile device 100 acquiresthe second (zero degree) gaze angle while the eye tracking function isexecuted, the mobile device 100 releases the mode where it has pausedthe content playback and resumes playing back the content as shown indiagram 405. Therefore, the mobile device 100 according to thisembodiment pauses, if the user does not gaze at the display unit 140playing back the content, the content playback without an additionaloperation, and then resumes playing it back if the user gazes at thedisplay unit 140 again.

If the mobile device 100 continues to acquire the non-zero gaze anglefor a certain period of time, when it displays the screen as shown indiagram 403, it turns off the display unit 140. Alternatively, themobile device 100 may turn off the display unit 140 and also terminatethe content playback function. As another alternative, the mobile device100 terminates the content playback function after a certain period oftime has elapsed from a time point that it turns off the display unit140. In addition, the mobile device 100 may terminate the eye trackingfunction at a time point when the display unit 140 is turned off or whenthe content playback function is terminated, thereby preventing electricpower consumption according to the activation of the camera unit 170.After that, if the mobile device 100 acquires the second gaze angle viaeye tracking, it enters a playback mode such as that shown in diagram405.

FIG. 5 illustrates user interface screens that support user functionsvia eye tracking, according to a further embodiment of the invention. Inthe embodiment, the mobile device 100 controls an e-book content viewfunction via eye tracking.

Referring to FIG. 5, the user selects an e-book content view function inthe mobile device 100. To this end, the mobile device 100 stores thee-book content in the storage unit 150 and also an application forplaying back it. If the user generates signals for selecting thecorresponding e-book content and playing it back, the mobile device 100plays it back on the screen as shown in diagram 501. In addition, themobile device 100 determines whether the e-book content view function isassociated with eye tracking. If it is associated with eye tracking, adetermination of whether the eye tracking mode is activated is made. Ifeye tracking is activated, the camera unit 170 acquires imagesperiodically or in real time via eye tracking.

If the mobile device 100 tracks the user's eye positions and identifiesthat the user views the display unit 140 at a preset gaze angle, itturns over the page as shown in diagram 503. For example, if the mobiledevice 100 acquires a gaze angle at which the user is looking at thebottom right corner portion on the display unit 140, it detects thisgaze angle detection as a command for turning the page to the next page.If the mobile device 100 continues to acquire this gaze angle for acertain period of time, it detects it as a command to continue turningpages.

Therefore, the mobile device 100 allows the user to conveniently turnpages of the e-book, via eye tracking, without an additional commandcorresponding to a touch action or an input signal. In someimplementations, mobile device 100 may adaptively alter the applicationperiod of eye tracking. That is, the mobile device 100 checks and storesthe number of pages turned over, and activates the camera unit 170 inorder to support eye tracking, for a certain period of time, before anestimated time point that an event for turning over pages will occur,according to the information. For example, if it takes an average of twominutes for the user to turn to the next page, the mobile device 100disables the camera unit 170 for a “camera disabling time” shorter thanthe average page turning interval. For instance, after an event forturning a page occurred, the camera is disabled and then enabled againafter 1 minute 30 seconds or 1 minute 40 seconds has elapsed from thetime point that the event occurred in order to support eye tracking. Ifthe page turning interval has gradually shortened, the mobile device 100may dynamically reduce the camera disabling time corresponding to theshortened interval, so that it can support the eye tracking function. Tothis end, the mobile device 100 may estimate the average page turninginterval via a preset number of pages. For example, the mobile device100 may limit the number of pages turned over to three, estimate theaverage page turning interval with respect to the three turned pages,and then define the camera disabling time to support the eye trackingfunction. This method prevents the camera unit 170 from wasting electricpower, compared with a state where the camera unit 170 remains enableddespite not executing the eye tracking function.

If the mobile device 100 displays the last page of the e-book content onthe display unit 140, it may automatically terminate the eye trackingfunction.

Meanwhile, if the mobile device 100 acquires a user's gaze angle forselecting the previous page, e.g., a gaze angle at which the user gazesat the bottom left corner portion on a page, it detects it as a commandfor turning the page to the previous page and then performs the pageturning operation to the previous page. In addition, the mobile device100 may also define the page turning speed according to the period oftime for which it acquires a gaze angle at which the user gazes at thebottom right or left portion on a page. That is, if a user gazes at thebottom right or left portion on a page for a certain period of time, themobile device 100 may set the page turning speed so that it graduallyincreases or is maintained over a preset value. Therefore, the user canrapidly search for a corresponding page. As described above, the mobiledevice 100 can control the page turning speed according to the period oftime for which it acquires a gaze angle for turning pages, and stopturning pages if it acquires a gaze angle that differs from the gazeangle for turning pages. The embodiment of FIG. 5 is particularlyadvantageous for helping a disabled or paralyzed person read an e-book.

FIG. 6 is a flow chart that describes a method for controlling a userfunction based on eye tracking, according to an embodiment of theinvention. At the start, the controller 160 (hereafter, “the controller”for brevity) supplies electric power to the components in the mobiledevice 100 (601). At this point, the controller is ready to execute userfunctions via eye tracking.

If the user generates a signal to execute a specific user function, themobile device 100 executes it. During this process, the controllerdetermines whether an eye tracking mode is set (603). For example, thecontroller identifies whether the user function is associated with eyetracking, and, if the user function is associated with eye tracking,whether the eye tracking mode is activated. If the eye tracking mode isnot set at step 603, the activated user function is controlled accordingto the input signal (605).

On the contrary, if the eye tracking mode is set at step 603, the cameraunit 170 is activated (607). The controller controls the camera unit 170to acquire images. After acquiring the images, the controller executesthe eye tracking function (609). That is, the positions of the user'seyes are identified as well as the gaze angle according to the eyepositions.

The controller performs a corresponding operation according to the gazeangle (611). For example, if the controller ascertains that the usergazes at the display unit 140 (hereafter, “the display”), according tothe acquired gaze angle, in a state where a content view function or acontent playback function has been executed, it maintains the executionof the content view function or the content playback function. That is,the controller supplies electric power to the display and controls thedisplay to display the content. On the contrary, if the controllerascertains that the user doesn't gaze at the display based on theacquired gaze angle, it controls the display to reduce the level ofbrightness or turns off the display, or pauses or terminates the contentplayback. If an e-book playback function has been executed, thecontroller controls the page turning operation of the e-book accordingto the acquired gaze angles.

If the controller acquires a gaze angle at which the user gazes at thedisplay during the reception of a call connection request message, itperforms an automatic call connection. This is described in detailreferring to FIG. 7. As shown in FIG. 7, if the mobile device 100receives a call connection request message from another mobile device,the controller is awakened from the sleep mode or displays a screeninforming that a call connection request message is received in an idlestate, as shown in diagram 701. The controller ascertains that the callconnection function is associated with eye tracking and activates thecamera unit 170. The controller activates the camera unit 170 for acertain period of time and identifies whether to identify a preset gazeangle from the acquired image. If the controller identifies a presetgaze angle from the acquired image, it performs an automatic callconnection as shown in diagram 703.

If the call connection request is ended while maintaining the executionof the eye tracking function, the controller may terminate the eyetracking function. The controller may execute the eye tracking functionfor only a preset period of time, so that a corresponding function isterminated when the preset period of time has elapsed. If a call isconnected and then the call function terminated, the controller mayterminate the eye tracking function. If a call is connected via eyetracking and the controller acquires a gaze angle at which the usergazes at the display after a certain period of time has elapsed from atime point that the call connection was established, the controller mayterminate the call connection. In addition, the controller may alsoestablish a call connection by matching information, acquired via a facerecognition procedure, as well as a gaze angle of the user's eyes, withthe user's preset information. For example, only if the controlleridentifies a face that was registered, as a mobile device owner's face,in the mobile device, it can acquire the gaze angle and establish a callconnection according to the acquired gaze angle.

After performing a corresponding operation according to the gaze angleat step 611, the controller determines whether an event for releasingthe eye tracking mode occurs (613). If so, the controller determineswhether a signal for terminating the procedure is input (615). Examplesof an event for releasing the eye tracking mode may be a case where thecontroller acquires a gaze angle at which the user doesn't gaze at thedisplay for a certain period of time or a case where a touch action oran input signal for releasing the eye tracking mode is generated.

On the contrary, if the controller ascertains that an event forreleasing the eye tracking mode has not occurred at step 613, it returnsto and proceeds with step 607. In addition, if the controller ascertainsthat a signal for terminating the procedure is not input at step 615, itreturns to and proceeds with step 603.

Although the embodiment just described includes a step for determiningwhether the eye tracking mode is activated, as an alternative, the eyetracking mode can be executed by default.

As described above, methods according to the invention can control userfunctions in a mobile device, such as altering the states or maintainingthe current state, via eye tracking, without requiring additionaloperations such as the application of touch actions or the generation ofinput signals, thereby facilitating use of the mobile device. Inparticular, if the invention is applied to mobile devices with largedisplays and typically held in one hand and operated by the other, usethereof is facilitated via eye tracking.

As described above, the method and the mobile device adapted thereto,according to the invention, can control efficiently and adaptively avariety of functions, based on eye tracking, so that users canconveniently use the mobile device.

Although not shown in the drawings, the mobile device may selectivelyfurther include various types of components, for example: a short-rangecommunication module for short-range communication; an interface fortransmitting/receiving data in a wireless or wired mode; an Internetcommunication module; and a digital broadcast module for receiving andreproducing broadcasts. With the spread of digital convergence, althoughit is unnecessary to list all the modifications of mobile devices inthis description, it will be easily appreciated to those skilled in theart that the other components equivalent to the above-listed componentsmay be further included to the mobile device according to the invention.Also, it will be appreciated that, according to specific applications,the mobile device may be implemented by omitting a particular componentor replacing it with other components.

The above-described methods implemented via controller 160 according tothe present invention can be implemented in hardware, firmware or viathe execution of software or computer code that can be stored in arecording medium such as a CD ROM, an RAM, a floppy disk, a hard disk,or a magneto-optical disk or computer code downloaded over a networkoriginally stored on a remote recording medium or a non-transitorymachine readable medium and to be stored on a local recording medium, sothat the methods described herein can be rendered in such software thatis stored on the recording medium using a general purpose computer, or aspecial processor or in programmable or dedicated hardware, such as anASIC or FPGA. As would be understood in the art, the computer, theprocessor, microprocessor controller or the programmable hardwareinclude memory components, e.g., RAM, ROM, Flash, etc. that may store orreceive software or computer code that when accessed and executed by thecomputer, processor or hardware implement the processing methodsdescribed herein. In addition, it would be recognized that when ageneral purpose computer accesses code for implementing the processingshown herein, the execution of the code transforms the general purposecomputer into a special purpose computer for executing the processingshown herein.

The terms or words described in the present description and the claimsshould not be limited by a general or lexical meaning, instead should beanalyzed as a meaning and a concept through which the inventor definesand describes the exemplary embodiments of the invention at his besteffort. Therefore, one skilled in the art will understand that theembodiments disclosed in the description and configurations illustratedin the drawings are only exemplary embodiments, instead there may bevarious modifications, alterations, and equivalents thereof to replacethe embodiments at the time of filing this application. Althoughexemplary embodiments of the invention have been described in detailhereinabove, it should be understood that many variations andmodifications of the basic inventive concept herein described, which maybe apparent to those skilled in the art, will still fall within thespirit and scope of the exemplary embodiments of the invention asdefined in the appended claims.

What is claimed is:
 1. A method comprising: identifying, at anelectronic device, a direction of a user's body part facing toward adisplay of the electronic device; determining a time period for whichthe direction of the user's body part is maintained; executing a firstfunction based at least in part on a determination that the time periodsatisfies a specified time period; and executing a second function basedat least in part on a determination that the time period does notsatisfy the specified time period.
 2. The method of claim 1, furthercomprising: prior to the direction being identified, activating an eyetracking mode based at least in part on a determination that a specifiedapplication is executed.
 3. The method of claim 1, wherein the executingof the first function comprises: adjusting a level of illumination ofthe display or a volume of an audio content to be presented at theelectronic device based at least in part on the time period.
 4. Themethod of claim 1, wherein the identifying comprises: detecting a gazeangle of a user's face as the user's body part.
 5. The method of claim4, wherein the user's face comprises one of more eyes, a nose, aeyebrow, a jaw line or a combination thereof.
 6. The method of claim 1,wherein the executing of the second function comprises: pausing orterminating execution of an application.
 7. The method of claim 1,wherein the executing of the first function comprises: moving an imageobject on the display based at least in part on the time period.
 8. Themethod of claim 1, wherein the executing of the first functioncomprises: maintaining a current screen of the display while thedirection toward the display.
 9. The method of claim 1, wherein theidentifying comprises: detecting presence of a face from the user's bodypart; automatically focusing on an iris in the face; and acquiring agaze of the iris.
 10. The method of claim 1, wherein the executing ofthe first function comprises: executing an application related to agraphic object on the display if the direction toward the graphicobject.
 11. An apparatus comprising: a camera unit; a display; and acontroller operatively coupled with the camera unit, the controllerconfigured to: identify a direction of a user's body part facing towardthe display; determine a time period for which the direction of theuser's body part is maintained; execute a first function based at leastin part on a determination that the time period satisfies a specifiedtime period; and execute a second function based at least in part on adetermination that the time period does not satisfy the specified timeperiod.
 12. The apparatus of claim 11, wherein controller is configuredto: manage a list of functions associated with eye tracking.
 13. Theapparatus claim 11, wherein the controller is configured to: reduce anillumination level of the display or inactivated the display if thedirection facing toward the display.
 14. The apparatus of claim 13,wherein the controller is configured to: inactivate the camera unit ifthe display is turned off or an application being executed at theapparatus is terminated.
 15. The apparatus of claim 11, wherein thecontroller is configured to: pause or terminate execution of anapplication if the direction facing toward the display.
 16. Theapparatus of claim 11, wherein the controller is configured to: changethe state of a touch panel of the apparatus based at least in part onthe direction.
 17. The apparatus of claim 11, wherein controller isconfigure to: control the camera unit to automatically focus on an iris,a nose, an eyebrow or a jaw line of the user's face to acquire thedirection.
 18. The apparatus claim 11, wherein the controller isconfigured to: execute the first function or the second function relatedto an object on the display if the direction indicates that the user'sface toward the object.
 19. The apparatus of claim 11, wherein thecontroller is configured to: execute the first function or the secondfunction to move a graphic object on the display based at least in parton the time period.
 20. A non-transitory machine-readable storage devicestoring instructions that, when executed by one or more processors,cause the one or more processors to perform operations comprising:identifying, at an electronic device, a direction of a user's body partfacing toward a display of the electronic device; determining a timeperiod for which the direction of the user's body part is maintained;executing a first function based at least in part on a determinationthat the time period satisfies a specified time period; and executing asecond function based at least in part on a determination that the timeperiod does not satisfy the specified time period.